A Constructive Algorithm for Feedforward Neural Networks With Incremental Training
نویسندگان
چکیده
We develop, in this brief, a new constructive learning algorithm for feedforward neural networks. We employ an incremental training procedure where training patterns are learned one by one. Our algorithm starts with a single training pattern and a single hidden-layer neuron. During the course of neural network training, when the algorithm gets stuck in a local minimum, we will attempt to escape from the local minimum by using the weight scaling technique. It is only after several consecutive failed attempts in escaping from a local minimum that will we allow the network to grow by adding a hidden-layer neuron. At this stage, we employ an optimization procedure based on quadratic/linear programming to select initial weights for the newly added neuron. Our optimization procedure tends to make the network reach the error tolerance with no or little training after adding a hidden-layer neuron. Our simulation results indicate that the present constructive algorithm can obtain neural networks very close to minimal structures (with the least possible number of hidden-layer neurons) and that convergence (to a solution) in neural network training can be guaranteed. We tested our algorithm extensively using a widely used benchmark problem, i.e., the parity problem.
منابع مشابه
A New Constructive Method to Optimize Neural Network Architecture and Generalization
In this paper, after analyzing the reasons of poor generalization and overfitting in neural networks, we consider some noise data as a singular value of a continuous function jump discontinuity point. The continuous part can be approximated with the simplest neural networks, which have good generalization performance and optimal network architecture, by traditional algorithms such as constructi...
متن کاملA constructive training algorithm for feedforward neural networks with ternary weights
We address the problem of training a special kind of Boolean feedforward neural networks, namely networks built from linear binary threshold units with ternary weights called majority units. We propose an original constructive training algorithm, inspired from both the approaches of the Tiling and the Upstart algorithms, and we give a simple proof of its convergence. Numerical experiments were ...
متن کاملConstructive Neural Networks: a Review
In conventional neural networks, we have to define the architecture prior to training but in constructive neural networks the network architecture is constructed during the training process. In this paper, we review constructive neural network algorithms that constructing feedforward architecture for regression problems. Cascade-Correlation algorithm (CCA) is a well-known and widely used constr...
متن کاملConstructive algorithms for structure learning in feedforward neural networks for regression problems
In this survey paper, we review the constructive algorithms for structure learning in feedforward neural networks for regression problems. The basic idea is to start with a small network, then add hidden units and weights incrementally until a satisfactory solution is found. By formulating the whole problem as a state-space search, we first describe the general issues in constructive algorithms...
متن کاملConstructive Neural Network Algorithms for Function Approximation Tasks
The generalization capability and training time of conventional neural networks depend on their architecture. In conventional neural networks, we have to define the architecture prior to training but in constructive neural network (CoNN) algorithms the network architecture is constructed during the training process. This paper presents an overview of CoNN algorithms that constructing feedforwar...
متن کامل